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ABSTRACT 

Using  a  unique  sample  of  college  students  at  a  service  academy,  this  paper  explores  whether  the 
testing  effect  can  be  applied  to  an  introductory  corporate  finance  course.  Prior  research  on  the 
testing  effect  has  been  used  non-quantitative  subject  matter.  We  introduce  the  testing  effect  to 
undergraduate  finance  and  find  evidence  that  administering  a  blind  pretest  at  the  beginning  of  the 
semester  improves  final  exam  performance  at  the  end  of  the  semester.  This  performance 
improvement  occurs  without  providing  feedback  to  students  or  instructors.  The  improvement  is 
largest  for  average  and  below  average  students. 
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INTRODUCTION 

Corporate  finance  courses  have  long  challenged  undergraduate  business  majors.  It  is  well  known 
that  many  students  dread  taking  finance.  The  quantitative  nature  of  finance  clearly  discourages  some. 
Anecdotes  abound  of  business  students  delaying  enrollment  in  finance  courses  as  long  as  possible. 
Deferring  enrollment  will  likely  exacerbate  student  difficulties  though  because  finance  material  is 
integrative.  Mastering  finance  material  requires  that  the  student  understand  a  number  of  fundamental 
concepts  from  accounting  and  economics. 

Given  these  challenges,  researchers  have  been  eager  to  identify  the  determinants  of  academic 
success  in  finance  and  economics  coursework.  Research  in  this  area  can  help  with  a  fundamental  problem 
facing  finance  educators  and  learners  alike  and  identify  the  optimal  learning  conditions  to  meet  these 
challenges.  Extant  research  looks  at  academic  factors,  such  as  student  entrance  exam  scores,  performance  in 
prerequisite  courses  and  grade  point  average  (GPA).  Studies  also  look  at  course  design  and  instructor  traits. 
Still  other  studies  focus  on  the  role  that  demographic  factors  such  as  gender  or  age  may  play. 

Some  of  these  research  findings  may  have  already  paid  dividends  in  the  classroom.  Test  data  taken 
from  the  major  field  test  in  business  shows  a  significant  improvement  in  finance  scores  over  time.  The 
major  field  test  measures  the  business-related  knowledge  of  seniors  in  undergraduate  business  programs. 
Each  year  the  Educational  Testing  Service  (ETS)  Comparative  Data  Guide  publishes  aggregate  scores  of  the 
core  business  areas  including  finance.  In  2000,  the  ETS  data  listed  finance  as  last  of  eight  business  topic 
areas.  By  2014  ETS  data  shows  marked  improvement  with  finance  ranked  fifth  in  nine  areas1. 

Further  gains  in  student  learning  may  require  the  application  of  recent  insights  in  the  psychology  of 
learning.  This  research  has  yielded  a  better  understanding  of  how  people  learn  and  what  facilitates  effective 
learning.  Some  of  these  findings  have  yet  to  be  systematically  applied  to  undergraduate  finance.  This  paper 
documents  the  application  of  one  such  insight  to  a  core  corporate  finance  course. 

In  this  study,  we  evaluate  the  pedagogical  effectiveness  of  pretests  within  the  context  of  an 
undergraduate  corporate  finance  course.  A  pretest  instrument  is  administered  to  undergraduate  finance 
students  on  the  first  day  of  class.  The  students  do  not  have  an  opportunity  to  study  course  material  before 
testing,  which  is  why  it  is  labeled  a  blind  pretest.  The  pretest  is  a  multiple  choice  test  with  questions  and 
answers  qualitatively  similar  to  those  found  on  the  final  exam.  Neither  the  students  nor  the  instructors  are 
provided  any  pretest  feedback  throughout  the  semester.  Performance  on  the  final  exam  is  then  evaluated 
and  compared  to  that  of  a  similar  control  group  that  did  not  take  the  pretest.  Including  a  robust  set  of 
academic  and  demographic  controls,  we  find  evidence  that  suggests  blind  pretesting  is  associated  with 
improved  final  exam  performance,  particularly  for  average  and  lower-performing  students. 

This  study’s  contributions  to  the  research  literature  are  three-fold.  The  findings  here:  (1)  confirm 
that  blind  pretesting  can  add  value  to  student  learning/ achievement  in  a  classroom  setting,  (2)  suggest  that 
blind  pretesting  can  add  value  in  a  corporate  finance  course,  an  academic  discipline  that  blends  soft  and 
hard  science,  and  (3)  indicate  that  average  and  lower-performing  students  may  benefit  disproportionately 
from  using  pretests. 

LITERATURE  REVIEW 

The  literature  review  will  provide  a  brief  synopsis  of  two  streams  of  research.  First,  we  summarize 
pretesting  as  a  learning  catalyst.  Second,  we  review  the  determinants  of  student  performance  in 
undergraduate  finance  that  we  will  use  as  controls  in  order  to  appropriately  isolate  any  potential  pretesting 
effect. 

The  Testing  Effect 


1  https://www.ets.org/ s/ mft/ pdf/ acdg  business.pdf 
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Tests  have  traditionally  been  used  to  assess  learning,  but  for  a  while  researchers  have  known  that 
testing  may  actually  facilitate  student  learning.  This  remarkable  discovery  is  labeled  the  testing  effect.  Few 
educators  utilize  the  testing  effect  however.  Anecdotal  evidence  suggests  that  within  the  finance, 
accounting  and  economics  community  pretests  are  rarely  used.  This  is  not  surprising  given  the  general  lack 
of  awareness  of  the  testing  effect.  Though  the  testing  effect  is  not  widely  known,  it  is  powerful  and  reliable 
in  a  large  number  of  contexts  and  has  been  verified  in  experimental  and  classroom  settings  (Roediger  and 
Karpicke,  2006a). 

Bjork  (1975)  stands  out  among  the  earliest  studies  to  speculate  on  why  testing  may  function  as 
learning  event.  What  is  clear  from  this  study  is  that  information  is  not  merely  accessed.  Instead,  the  act  of 
retrieving  information  alters  how  that  information  is  stored  in  memory.  The  process  of  retrieving 
information  alters  the  accessibility  of  this  information,  making  it  more  recallable  in  the  future. 

Taking  a  test  may  be  a  more  effective  learning  strategy  over  long  periods  of  time  than  simply 
reviewing  material  a  second  time.  In  experiments  using  word  list  recall,  Hogan  and  Kintsch  (1971)  and 
Thompson,  Wenger,  and  Bartling  (1978)  each  find  that  a  testing  approach  is  superior  to  a  pure  study 
approach  for  long-term  retention.  Roediger  and  Karpicke  (2006b)  compare  the  studying  of  a  prose  passage 
once  and  taking  a  test  versus  studying  the  same  passage  twice.  They  find  that  the  first  strategy  generates 
better  recall  when  the  time  delay  between  the  pretest  and  the  recall  test  is  more  than  one  day. 

Tests  that  appear  to  be  unsuccessful  do  not  necessarily  impede  learning.  On  the  contrary,  failed 
attempts  at  retrieving  information  can  actually  help  the  learning  process.  Kornell,  Hays  &  Bjork  (2009) 
conduct  experiments  that  ensure  learners  will  be  unsuccessful  at  retrieving  information.  Using  a  variety  of 
test  content  and  answer  formats,  the  authors  find  that  unsuccessful  tests  assist  future  learning  rather  than 
hinder  it.  Moreover,  learning  occurs  even  when  no  corrective  feedback  is  provided  to  the  student  (Carrier 
&  Pashler,  1992). 

Some  educators  have  used  pretests  in  a  formative  sense  to  identify  the  state  of  student's  knowledge 
at  the  beginning  of  a  course.  Formative  evaluation  can  be  quite  effective  to  facilitate  learning  if  the 
instructor  is  willing  to  adapt  and  make  real-time  changes  to  the  course.  On  a  least  one  occasion,  the  efficacy 
of  this  approach  has  been  demonstrated  in  teaching  finance.  Heinfeldt  and  Wolf  (2002)  introduce 
formative  evaluation  to  an  introductory  undergraduate  finance  course.  The  use  of  pre-  and  post-tests 
provides  beneficial  information  to  finance  instructors  and  assists  students  in  learning  the  material. 

While  formative  evaluation  can  indirectly  affect  student  learning,  pretests  can  also  be  used  to 
directly  affect  student  learning  by  harnessing  the  testing  effect.  The  testing  effect  is  powerful  and  reliable 
but  not  widely  used  in  the  classroom.  Research  has  confirmed  the  power  of  classroom  tests  but  the  efficacy 
of  the  testing  effect  has  not  been  tested  in  a  finance  setting.  This  study  is  arguably  the  first  to  do  so.  This  is 
important  since  the  testing  effect  has  been  generally  evaluated  outside  the  hard  sciences.  Finance  bridges 
the  gap  between  hard  and  soft  science.  Therefore,  a  classroom  evaluation  of  the  testing  effect  could 
potentially  benefit  the  teaching  of  accounting,  economics  and  other  disciplines  that  are  both  hard  and  soft 
science. 

Determinants  of  Performance  in  Undergraduate  Finance 

Researchers  have  investigated  a  number  of  factors  that  may  influence  student  performance  in 
undergraduate  finance  and  economics.  Bellico  (1972)  finds  that  freshman  and  sophomore  GPA  is 
predictive  of  performance  in  economics.  Transferring  from  community  college  is  negatively  related  to 
performing  well.  Simpson  and  Sumrall  (1979)  is  perhaps  the  first  study  to  systematically  identify  the 
determinants  of  student  performance  within  an  introductory  finance  course.  Age  and  performance  in  prior 
economics  classes  is  positively  related  to  performance  but  transferring  from  another  school  and  increasing 
work  hours  are  associated  with  worse  grades.  Perhaps  the  best  predictor  of  performance  in  introductory 
finance  is  prior  performance  in  accounting  (Shaffer  and  Calkins,  1980). 
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The  controls  used  in  this  study  are  mostly  found  in  two  large  sample  studies  that  each  focus  on 
student  performance  in  undergraduate  finance.  Borde,  Byrd  and  Modani  (1998)  look  at  performance  in 
introductory  finance  and  find  females  and  community  college  transfer  students  underperform.  GPA, 
performance  in  accounting  and  fewer  hours  of  employment  are  associated  with  better  performance.  In  a 
follow-up  study,  Terry  (2002)  finds  significant  performance  differences  linked  to  GPA,  major,  gender, 
performance  in  prerequisite  courses,  the  student's  major  and  whether  the  course  took  place  in  summer. 

INSTITUTIONAL  AND  SAMPLE  DESCRIPTION 

This  study  expands  the  testing  effect  literature  while  accounting  for  the  performance  determinants 
in  finance.  Here  we  analyze  the  effectiveness  of  a  blind  pretest  on  the  performance  of  students  in  a 
corporate  finance  course  at  the  US  Air  Force  Academy.  As  the  first  required  finance  course  for  these 
students,  the  course  analyzed  is  simultaneously  an  introductory  finance  course,  akin  to  Terry  (2002),  and  an 
intermediate  course,  analogous  to  Fields  (2013),  in  that  it  employs  a  graduate-level  textbook  and 
corresponding  approach.  Final  exam  scores  are  compared  between  students  who  took  a  blind  pre-test  at  the 
beginning  of  the  semester  with  those  who  did  not,  controlling  for  a  variety  of  factors  that  previous  studies 
have  found  are  associated  with  performance  in  related  finance  courses. 

Before  explaining  the  sample  details,  it  is  useful  to  understand  that  the  US  Air  Force  Academy 
presents  a  conducive  environment  for  pedagogical  studies.  Its  purpose  is  similar  to  that  of  the  US  Military 
Academy  or  the  US  Naval  Academy.  Upon  graduation,  cadets  receive  not  only  a  four  year  degree  but  are 
also  commissioned  as  officers  in  the  Air  Force.  Given  this,  the  institution  has  multi-faceted  goals  that 
include  not  only  academic  achievement  but  also  military  leadership,  athletic  accomplishment  and  character 
development.  This  drives  both  the  system’s  treatment  of  students  and  students’  individual  decisions  about 
allocating  their  time.  Students  must  perform  at  a  satisfactory  level  in  each  dimension  to  successfully 
graduate. 

This  institution  takes  many  steps  to  ensure  unbiased  treatment  of  officer  candidates  as  they  make 
progress  toward  each  institutional  goal.  Students  in  a  class  all  begin  their  time  at  the  Air  Force  Academy 
together  and  in  lock-step  complete  a  four-year  program.  The  four  year  program  is  never  shortened  and 
extended  only  in  rare  circumstances.  Once  a  cohort  of  cadets  starts  the  program,  there  are  no  additional 
student  transfers  from  community  colleges  or  other  institutions.  These  cadets  are  prohibited  from  living  off 
campus  and  live  in  one  of  two  dormitories.  Each  student  is  assigned  to  one  of  forty  'squadrons'  made  up  of 
approximately  100  students  representing  all  four  classes  and  virtually  every  possible  demographic 
dimension.  There  are  no  academic  fraternities,  sports  sororities  or  religion-based  dormitories  that  could 
serve  as  factors  that  systematically  bias  our  results. 

The  overall  academic  experience  is  remarkably  consistent  from  student  to  student.  Students  select 
an  academic  major  but  cannot  choose  their  instructors  or  semester  course  schedules.  All  students 
experience  a  very  similar  core,  or  general,  curriculum  for  the  first  two  years  before  beginning  their  majors 
courses.  Each  student  takes  from  five  to  seven  courses  in  any  given  semester,  with  the  average  converging  on 
six  courses  as  the  students  approach  their  junior  year  (96%  of  our  sample)  and  senior  year  (the  other  4%  of 
our  sample).  They  are  randomly-assigned  to  class  periods  and  instructors  so  that  all  students  can  attend  class 
yet  still  have  time  to  meet  at  a  common  time  with  their  flying  team,  military  unit,  or  sports  team.  Outside  of 
the  classroom,  all  students  have  military-  and  athletic-related  commitments.  Depending  on  the  position  or 
team,  these  commitments  can  vary  substantially.  There  are  no  concessions  in  a  student’s  academic 
requirements  because  of  these  roles  however.  Overall,  the  unique  academic  setting  of  the  US  Air  Force 
Academy  makes  it  difficult  to  argue  that  any  system-induced  in-classroom  or  beyond-the-classroom  bias 
affects  the  results  of  a  study  such  as  this  one. 

As  for  the  specific  study  design,  this  analysis  spans  two  academic  years  and  includes  results  from 
spring  2014  and  spring  2015  semesters.  The  entire  sample  includes  197  students  across  10  sections  of  a 
corporate  finance  course  taught  by  two  instructors.  The  spring  2014  students  represent  the  control  group 
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and  did  not  take  a  blind  pre-test  for  this  course.  Since  testing  can  facilitate  learning  not  only  in  cued  recall, 
free  recall  and  short-answer  tests,  but  also  in  a  multiple  choice  format  (Bjork,  Soderstrom  and  Little,  2015), 
the  treatment  group  received  a  15-minute,  17-question  multiple  choice  pre-test  on  lesson  two  of  the 
semester.  The  questions  were  representative  of  the  final  exam  for  this  particular  course.  We  label  this 
instrument  the  blind  pre-test.  It  is  blind  because  these  students  have  not  been  exposed  to  this  corporate 
finance  material  at  any  point  in  the  in  the  two-and-one-half  years  prior  to  taking  this  course.  We  posit  its 
effectiveness  based  on  the  findings  of  Kornell,  Hays,  and  Bjork  (2009),  who  determine  that  the  testing  effect 
can  be  valid  even  if  students  are  not  exposed  beforehand  to  the  material  over  which  they  are  tested. 

Some  demographic  factors  identified  in  prior  works  are  not  included  in  this  study  given  the  unique 
nature  of  the  educational  institution  and  related  student  sample.  For  example,  age  is  not  specified  here  in 
this  model.  The  student  population  is  tightly  clustered  since  by  law  newly  enrolled  students  must  be  no 
older  than  23  years  old.  Outside  employment  is  also  not  considered  in  this  paper  since  employment  is 
expressly  forbidden  by  the  institution.  Additionally,  transferring  from  a  community  college  or  other 
educational  institution  is  not  controlled  for  in  this  paper  since  it  is  not  possible  within  the  highly  controlled 
environment  of  a  service  academy. 

Students  were  not  allowed  to  transcribe,  photograph,  or  keep  the  blind  pre-test.  The  two  instructors 
administering  the  pretest  informed  the  students  that  they  were  not  expected  to  know  the  tested  material 
and  that  these  questions  were  representative  of  the  final  exam  questions  that  the  students  should  expect  to 
answer  by  the  end  of  the  course.  Instead  of  recommending  memorization,  the  instructors  told  the  students 
that  it  was  worth  their  time  to  attempt  to  answer  the  questions  correctly.  The  instructors  collected  the 
completed  pretests  which  were  then  filed  away.  It  is  important  to  note  that  the  instructors  did  not  grade  the 
blind  pretests.  Doing  so  would  have  permitted  instructors  to  make  adjustments  to  the  course  by  under-  or 
over-emphasizing  certain  content.  This  approach  was  chosen  to  isolate  any  potential  testing  effect  and  is 
consistent  with  Carrier  and  Pashler  (1992)  who  find  that  students  do  not  need  to  receive  feedback  for 
testing  to  facilitate  learning. 

The  classroom  environment  and  approach  was  nearly  identical  for  both  the  control  and  treatment 
groups.  The  only  observable,  material  differences  between  semesters  we  identify  are  (a)  the  student 
populations  and  (b)  the  current  events  instructors  subjectively  introduced  to  their  classrooms.  The  same  two 
instructors  taught  both  the  control  and  treatment  semesters.  Both  instructors  in  this  study  have  terminal 
degrees  in  the  areas  of  finance,  and  both  have  been  teaching  at  the  same  institution  for  at  least  five  years  (10 
semesters)  before  the  spring  2014  semester.  Each  instructor  had  taught  this  course  for  multiple  years  and 
approached  the  classroom  and  course  content  just  as  they  had  done  in  prior  years.  The  syllabus  of  course 
material  for  both  semesters  was  identical  as  was  the  textbook  edition,  the  lecture  slides,  in-class  workshop 
exercises,  and  mid-semester  exams.2  Quizzes  across  groups  were  identical  in  concept  and  format,  however, 
for  workout  problems  the  instructors  changed  some  of  the  numbers  to  test  computational  abilities  instead 
of  whether  the  student  had  received  a  copy  of  a  prior  semester’s  quizzes. 

Using  the  spring  2014  and  spring  2015  semesters  as  the  control  and  treatment  group  provides  a 
clean  experimental  design.  Using  semesters  prior  to  spring  2014  would  have  increased  the  sample  size  but 
would  have  introduced  multiple  sources  of  noise.  The  course  switched  from  a  fall-only  offering  to  a  spring- 
only  offering  in  2013.  Thus,  beginning  the  study  in  spring  2014  allowed  for  a  one-year  period  of  transition 
from  fall  to  spring.  Additionally,  this  course  is  not  offered  in  the  summer  at  the  Air  Force  Academy, 
prohibiting  any  noisiness  that  could  result  if  we  were  to  compare  the  effects  of  condensed  offerings  on  these 
results.  Another  reason  for  this  two-semester  analysis  includes  the  stability  of  the  49-question  multiple- 
choice  final  exam,  which  was  kept  identical  for  the  two  semesters  studied.  Its  contents  have  never  been 


2  Exams  in  this  course  are  not  returned  permanently  to  students.  Instead,  students  can  review  them  in  the  instructors’ 
offices  at  any  time— that  is,  students  have  visitation  rights  but  not  custody.  On  the  other  hand,  students  do  receive 
their  quizzes  permanently. 
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released  publicly.  Finally,  as  noted  before,  we  evaluate  spring  2014  and  spring  2015  due  to  their  identical 
course  content  and  team  of  terminabdegree  holding  instructors. 

METHOD  OF  ANALYSIS 

A  two  period  model  is  developed  in  which  the  corporate  finance  final  exam  score  measured  at  the 
end  of  the  semester  is  a  function  of  the  modeled  characteristics  at  the  beginning  of  the  semester.  The 
variable  of  interest,  Pretest,  is  a  binary  variable  equal  to  one  if  the  student  took  the  pretest  and  zero 
otherwise.  The  discussion  below  identifies  the  vector  of  controls  used.  Equation  (1)  provides  the 
conceptual  framework. 

Final  Exam  = 

/(Pretest,  GPA,  IC,  Male,  Management  Major,  Senior,  Acct  Prereq,  Instructor)  ^ 

Consistent  with  prior  studies,  we  use  incoming  GPA,  or  the  student’s  cumulative  GPA  when 
entering  this  course,  as  a  proxy  for  student  ability.3  Prior  studies  have  addressed  the  significant  presence  of 
intercollegiate  athletes  at  this  service  academy  (see  Payne,  Bredthauer,  Martin,  and  Merrell,  2014).  We  use 
a  binary  variable  where  one  signifies  intercollegiate  status  and  zero  an  absence  of  this  status  (IC).  Gender  is 
captured  with  a  binary  variable  where  male  equals  one  and  female  zero.  Since  prior  studies  have  assessed  the 
effect  of  academic  major  on  course  grades,  we  do  the  same.  A  binary  variable  is  used  and  is  equal  to  one  if 
the  student’s  major  is  Management;  zero  otherwise.  The  non-Management  majors  typically  include  students 
majoring  in  Economics. 

The  binary  variable,  Senior,  equals  one  if  the  student  is  a  senior  taking  the  course  and  zero  if  a 
junior.  A  small  minority  of  the  students  are  seniors  who  experience  a  one  year  delay  between  the  end  of 
their  prerequisite  accounting  course  and  the  subject  corporate  finance  course.  We  control  for  this 
difference  among  individuals  because  of  the  obvious  lag  between  the  prerequisite  material  and  the  finance 
material  for  the  seniors.  Also,  there  could  be  an  additional  and  compounding  “senioritis”  effect,  as  the 
senior  cadets  are  mere  days  from  graduation.  These  seniors  already  know  what  their  first  jobs  and  locations 
will  be  after  graduation,  and  (observationally)  sometimes  approach  difficult  courses  in  their  final  semesters 
as  pass/fail  propositions.  Given  these  realities,  we  do  not  anticipate  a  positive  relationship  between  senior 
status  and  final  exam  performance.  Financial  Accounting  (Acct  Prereq )  is  the  only  prerequisite  course  for 
the  subject  corporate  finance  course.  This  variable  represents  the  GPA  in  this  course  as  measured  on  a  4.0 
scale.  Lastly,  instructor  fixed  effects  are  captured  with  a  final  binary  variable. 

RESULTS 

Descriptive  Results 

Table  1  shows  the  summary  statistics  for  the  student  population.  Panel  A  depicts  the  entire  sample  of  197 
students.  Final  Exam  measures  the  integer  number  of  correct  answers  on  the  49-question  final  exam  in  the 


3  The  incoming  cumulative  GPA  measure  at  the  US  Air  Force  Academy  is  a  remarkably  “apples  to  apples”  measure  of 
student  academic  performance  in  this  unique  environment.  This  is  the  case  because  the  students  have  previously  taken 
two  and  one-half  years  of  a  core,  or  general,  curriculum  that  is  over  80%  common  across  all  cadets  and  over  93% 
common  for  cadets  with  the  same  academic  major  (e.g.,  Management  major).  They  have  also  taken  2-4  common 
courses  in  their  major,  to  include  the  Financial  Accounting  prerequisite.  Finally,  the  institution  and  Management 
Department  take  deliberate  measures  to  control  so-called  “grade  inflation”  as  well  as  the  cross-sectional  variance  of 
different  instructors  (i.e.,  standardized  curriculum,  group  instructor  grading  parties,  and/or  grading  curves  within  an 
individual  instructor’s  students). 
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corporate  finance  course.  Its  mean  is  approximately  29.5  questions  correct  with  a  standard  deviation  of 
almost  6  questions.  Measured  on  a  4.00  scale,  the  average  incoming  GPA  for  these  students  is  2.65.  We 
note  that  approximately  39  percent  of  our  student  sample  is  intercollegiate  athletes  and  84  percent  of  our 
sample  is  male.  The  average  grade  for  all  students  in  the  Financial  Accounting  prerequisite  course 
approaches  a  B  (2.94)— again,  measured  on  a  4.00  scale.  Approximately  97  percent  of  these  students  are 
Management  majors.  The  sample  student  population  consists  of  96%  juniors  who  took  the  accounting 
prerequisite  during  the  immediately  preceding  fall  semester.  The  balance,  4%,  are  seniors  (Senior). 

We  next  check  for  systematic  differences  between  the  control  and  treatment  groups  by  subdividing 
this  overall  sample  into  those  who  did  not  take  the  blind  pretest  (spring  2014)  and  those  who  did  (spring 
2015).  The  final  column  shows  the  differences  in  mean  sub-sample  values  and  results  of  a  T-test  of  these 
differences.  We  find  evidence  that  those  who  took  the  blind  pretest  answered  a  little  more  than  three 
questions  correctly  on  the  final  exam  than  did  those  who  did  not  take  the  blind  pretest.  Also,  there  is  a 
significantly  higher  proportion  of  intercollegiate  athletes  in  the  control  group  (45  percent)  than  in  the  blind 
pretest  group  (31  percent).  Otherwise,  there  is  no  statistical  difference  in  these  groups’  average  incoming 
GPA,  the  gender  composition,  the  prerequisite  accounting  course  grade,  the  proportion  of  Management 
majors,  or  the  proportion  of  senior  cadets. 

Since  one  contribution  of  this  paper  to  the  existing  literature  on  pretesting  is  the  analysis  of  which 
students  derive  benefits  from  pretesting,  Table  1,  Panel  B  further  subdivides  the  whole  sample  into  thirds, 
or  terciles,  by  ability.  Once  again,  we  use  incoming  GPA  as  our  measure  of  ability  and  divide  each  group- 
control  and  treatment— into  thirds  by  their  incoming  GPAs.  The  one-third  of  students  with  the  highest 
GPAs  represent  the  “high  performing”  tercile,  the  one-third  with  the  lowest  GPAs  represent  the  “low 
performing”  tercile,  and  the  middle  one-third  are  the  “mid  performing”  tercile.  The  intent  here  is  to  discern 
whether  blind  pretesting  asymmetrically  benefits  one  tranche  of  students  relative  to  another. 

In  the  aggregate,  Table  1,  Panel  B  shows  that  the  incoming  GPA  is  positively  related  to  the 
corporate  finance  final  exam  performance  and  the  grade  in  the  prerequisite  accounting  course.  The  mean 
values  for  each  of  these  three  measures  decrease  monotonically  from  tercile  one  to  tercile  three.  These 
patterns  are  consistent  in  both  the  control  group  (No  Pretest  (2014))  and  the  treatment  group  (Pretest 
(2015)).  Overall  non-Management  majors  and  seniors  tend  to  decrease  as  a  proportion  of  the  sub-samples 
moving  from  higher-to-lower  performing  terciles,  there  is  no  clear  monotonic  relationship  when  broken  out 
by  control  and  treatment  groups. 

Because  we  ultimately  want  to  test  whether  the  blind  pretest  induces  any  difference  between 
otherwise  similar  students,  the  last  column  in  Panel  B  once  again  tests  for  differences  between  these 
performance  terciles.  While  there  are  no  consistent  differences  between  the  control  and  treatment  groups 
across  all  three  terciles,  some  insights  emerge.  For  high  performers,  final  exam  scores  are  not  significantly 
different  for  the  group  with  the  blind  pretest;  this  is  not  the  case  for  the  lower  two  terciles,  as  the  blind 
pretest  students  experience  significantly  higher  mean  scores  on  the  final  exam.  Pretested  students  in  tercile 
one  are  disproportionately  female,  have  lower  average  prerequisite  accounting  grades,  and  are  significantly 
more  senior  than  the  control  sample.  Besides  the  aforementioned  final  exam  grade  difference,  tercile  two 
has  more  seniors  in  the  control  group  than  the  treatment  group,  which  has  none.  Finally,  in  tercile  three 
the  pretested  group  has  a  higher  incoming  GPA  and  lower  number  of  intercollegiate  athletes  than  the 
control  group. 

Overall,  these  univariate  results  suggest  that  blind  pretesting  is  associated  with  higher  final  exam 
scores  in  this  sample  of  students.  Furthermore,  the  performance  increase  occurs  primarily  in  the  lower- 
performing  subset  of  this  student  group. 

Pretesting  Matters 

Given  the  statistical  differences  between  the  control  and  treatment  sub-samples  in  Table  1,  we  must 
address  these  issues  before  reaching  any  conclusions  about  the  value  of  blind  pretesting  in  an  this 
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undergraduate  corporate  finance  course.  We  therefore  use  regression  analysis  to  disentangle  any  potential 
blind  pretest  effects  from  other  drivers  of  final  exam  performance. 

Before  presenting  the  regression  results,  it  is  insightful  to  discuss  the  correlations  between  the 
variables  in  our  analysis  and  their  impact  on  the  subsequent  regression  analysis.  Including  highly-correlated 
variables  as  independent  variables  in  regression  analysis  can  lead  to  erroneous  conclusions.  Table  2  presents 
these  pairwise  correlation  results  among  our  variables.  In  addition  to  the  variables  shown  in  Table  1,  Table 
2  includes  a  binary  variable  Pretest,  which  is  one  for  a  spring  2015  observation  and  zero  for  a  spring  2014 
observation.  Notably,  high  positive  correlation  exists  in  all  three  pair  permutations  of  Final  Exam,  GPA,  and 
Accounting  Prereq,  ranging  from  0.45  to  0.62.  Since  the  Final  Exam  score  represents  the  dependent  variable, 
the  methodological  concern  would  be  including  both  GPA  and  Accounting  Prereq  concurrently.  We  therefore 
present  models  that  both  include  and  exclude  the  accounting  prerequisite  grade.  Other  pairwise  correlation 
values  do  not  cause  as  much  concern,  as  each  measures  at  most  0.26  in  absolute  value.  We  use  these  others 
simultaneously  in  our  regression  models. 

Table  3  shows  the  regression  model  results  for  the  whole  sample.  The  coefficients  for  the  first 
model  represent  the  average  change  in  the  final  exam  score,  all  else  held  constant:  having  taken  a  blind 
pretest  (Pretest);  for  a  one  grade  point  change  in  incoming  GPA;  when  having  intercollegiate  athlete  (1C) 
status;  if  a  male;  as  a  Management  major;  when  taking  corporate  finance  as  a  final  semester  senior  (Senior); 
and  for  a  one  grade  point  change  in  the  financial  accounting  prerequisite  grade  (Accounting  Prereq).  The 
second  model  omits  the  financial  accounting  prerequisite  grade.  Both  models  control  for  any  instructor 
fixed  effects,  since  we  can  assume  that  the  instructors  differ  in  their  individual  approaches  given  differences 
in  education,  experiences,  personal  style  and  other  factors. 

If  there  are  no  meaningful  systematic  differences  between  the  spring  2014  and  spring  2015 
semesters— and  as  described  in  the  prior  section,  we  know  nothing  to  speak  of— then  these  results  suggest 
that  a  blind  pretest  is  associated  with  improved  final  exam  scores.  This  result  occurs  after  controlling  for 
factors  known  to  be  correlated  with  academic  performance,  such  as  past  GPA,  gender,  academic  major,  lag 
between  prerequisite  course  and  follow-on  course,  and  prerequisite  performance.  Additionally,  both  models 
indicate  that  that  final  exam  performance  is  significantly  and  positively  related  to  incoming  GPA  (i.e., 
ability).  In  the  first  model,  which  includes  the  accounting  prerequisite,  we  also  unsurprisingly  find  that 
performance  in  this  prerequisite  is  significantly  and  positively  related  to  the  corporate  finance  final  exam 
performance,  despite  this  variable’s  relatively  high  correlation  with  another  regressor,  GPA.  Consistent  with 
previous  findings,  we  see  no  effect  on  final  exam  performance  due  to  gender  or  academic  major. 
Intercollegiate  and  Senior  status  has  no  significant  statistical  relationship  with  final  exam  performance  in 
this  sample. 

Pretesting  Matters  -  Asymmetric  Effects 

Given  the  authors’  admitted  initial  skepticism  about  the  efficacy  of  a  blind  pretest  in  corporate 
finance,  it  is  quite  interesting  that  a  “preview”  of  the  calculations  and  concepts  that  students  will  experience 
in  a  corporate  finance  course  appears  to  generate  significant  learning  value.  However,  perhaps  most 
interesting  for  us  as  educators  is  whether  this  blind  pretest  value  is  equally  distributed  among  the  student 
population.  That  is,  does  a  blind  pretest  lift  all  (learning)  boats?  This  question  is  addressed  in  Table  4.  It 
shows  the  same  regression  results  as  Table  3  performed  for  each  of  the  three  separate  performance  terciles. 
As  before,  we  divided  students  into  three  terciles  by  sorting  them  according  to  their  incoming  GPA.  Tercile 
one  (three)  includes  the  highest-  (lowest-)  performing  one-third  of  the  students  as  measured  by  GPA.  We 
label  these  sub-groups  High  Performers  and  Low  Performers,  respectively.  The  middle  tercile,  tercile  two, 
has  an  original  label:  the  Mid  Performers. 

Using  the  same  variable  definitions  as  in  Table  3,  some  observations  immediately  stand  out.  First, 
in  contrast  to  the  whole  sample  findings,  for  high  performing  students  the  positive  impact  of  pretesting  is 


not  as  readily  apparent.  The  effect  is  insignificant  in  one  specification  and  barely  significant  at  conventional 
levels  in  another.  As  expected,  incoming  GPA  is  significant  for  the  high  performers;  and  our  priors  about 
any  lag  or  “senioritis”  effect  comes  to  the  fore  in  this  high  performing  tercile  (i.e.,  the  significantly  negative 
value  on  Senior).  That  is,  regardless  of  having  a  blind  pretest,  intercollegiate  status,  a  certain  gender, 
academic  major,  or  prerequisite  accounting  grade,  the  only  significant  explanatory  variables  for  the  final 
exam  grade  for  these  high  performers  is  their  incoming  GPA  and  whether  they  are  second-semester  seniors. 
Our  interpretation:  those  who  have  historically  done  well  will  continue  to  do  well  on  this  finance  final 
exam,  unless  they  have  time  to  forget  prerequisite  material  and/ or  perceive  no  incentive  to  perform  at  their 
highest  level  (read:  final  semester  seniors).  In  summary,  a  blind  pretest  has  a  debatable  statistical  effect  on 
these  high-performers’  final  exam  achievements. 

On  the  other  hand,  a  blind  pretest  is  associated  with  significantly  higher  final  exam  performance  in 
both  the  lower  two  tercile  performers,  across  all  model  specifications.  In  fact,  on  average,  experiencing  a 
blind  pretest  accompanies  seven  to  eight  percent  increase  in  final  exam  performance  for  these  mid  and 
lower  performing  students.  This  result  approaches  the  ten  percent  increase  seen  in  Bjork,  et.  al.  (2015). 
However,  the  result  here  occurs  using  a  multiple-choice  final  exam  that  consists  of  many  quantitatively- 
based  workout  problems.  It  encourages  those  of  us  seeking  to  enhance  the  academic  performance  of  at-risk, 
lower-performing  students,  particularly  in  subjects  that  are  more  quantitative  in  nature. 

While  the  mid  performing  tercile  essentially  mimics  the  findings  of  the  whole  sample,  ignoring  the 
constant,  the  results  in  the  low  performing  tercile  are  somewhat  confounding.  For  starters,  final  exam 
performance  is  not  statistically  related  to  GPA.  In  fact,  the  coefficient  estimate  is  negative,  which  belies 
prior  studies,  our  whole  sample  results,  and  intuition.  Perhaps  an  explanation  for  this  outcome  lies  in  the 
fact  that  while  not  significant  at  conventional  levels,  both  the  academic  major  (Management)  and 
prerequisite  (Acct  Prereq)  coefficients  are  positively-signed  and  have  p-values  of  0.105  and  0.115,  respectively. 
That  is,  for  students  who  have  struggled  academically  in  general— that  is,  they  have  low  GPAs  relative  to 
their  peers— final  exam  performance  is  not  related  to  their  overall  level  of  struggle.  Instead,  their  final  exam 
performance  in  a  particular  class  is  more  related  to  their  demonstrated  ability  in  that  course’s  prerequisite(s) 
or  courses  more  central  to  their  choice  of  academic  major.  Needless  to  say,  the  instructors  in  this  corporate 
finance  course  are  reflecting  deeply  on  the  methods  used  in  spring  2015,  as  they  are  clearly  associated  with 
improved  final  exam  performance  for  lower  level  students. 

We  conditionally  accept  these  findings,  knowing  that  the  testing  effect  associated  with  the  blind 
pretest  is  not  the  only  potential  explanation  for  the  enhanced  final  exam  performance.  It  is  possible  (and 
we’d  sure  hope)  that  our  teaching  effectiveness  improved  seven  or  eight  percent  year-over-year!  Yet,  based  on 
our  course  and  instructor  feedback,  this  explanation  is  doubtful.  There  may  be  an  omitted  variable  that 
better  explains  our  findings.  Other  systematic  differences  between  spring  2014  and  spring  2015  may  exist 
that  we  have  not  captured  (e.g.,  overall  mood  of  the  student  body,  the  growth  of  CNBC,  record  high  stock 
market  levels,  super  novas,  extra  eclipses,  global  warming,  a  break  in  the  Colorado  drought,  etc.).  At  this 
point  we  are  not  aware  of  any  systematic  differences  in  the  spring  2015  semester  that  would  have  directly 
and  materially  explained  this  increase  in  these  students’  final  exam  performance. 

CONCLUSIONS  AND  FUTURE  RESEARCH 

Using  a  sample  of  just  under  200  students  over  the  course  of  two  semesters,  we  find  that  taking  a 
blind  pretest  is  associated  with  significantly  higher  performance  on  the  final  exam  in  a  required  corporate 
finance  course  at  the  US  Air  Force  Academy.  At  this  institution  many  sources  of  variability  found  at  other 
institutions  are  not  present.  In  this  unique  environment  we  identify  a  real-world  (not  experimental) 
improvement  in  final  exam  performance  among  students  who  take  a  blind  pretest  at  the  beginning  of  a 
required  corporate  finance  course.  This  blind  pretest  does  not  explicitly  test  knowledge  presumed  from 
prerequisite  courses  but  instead  provides  final  exam  representative  questions  for  the  finance  course  students 
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are  about  to  take.  This  performance  delta  remains  robust  after  controlling  for  previously-determined 
predictive  factors,  such  as  student  GPA,  intercollegiate  athlete  status,  gender,  academic  major,  time  lag  since 
a  prerequisite  course,  and  performance  in  a  perquisite  course. 

Perhaps  most  interestingly,  we  find  asymmetric  effects.  After  subdividing  the  students  into  three 
groups  based  on  previous  overall  academic  performance,  we  find  that  the  blind  pretest  effect  does  not  hold 
universally  across  the  groups.  Instead,  the  positive  effect  is  concentrated  among  the  lower  two-thirds  of 
students.  This  result  suggests  that  blind  pretesting,  or  showing  key  course  concepts  in  an  exam  question 
form,  could  enable  these  mid-  and  lower-performing  students  to  see  the  areas  upon  which  to  focus  their 
intellectual  energy  over  the  course  of  the  semester. 

Pretesting  has  become  an  area  of  substantial  pedagogical  interest  among  researchers  and  educators 
alike.  The  so-called  testing  effect  has  broad  implication  to  education  as  educators  continually  endeavor  to 
determine  what  pedagogical  techniques  will  better  reach  students.4  This  study  contributes  to  the  analysis  of 
pretesting  as  a  pedagogical  technique  to  enhance  academic  performance. 

As  with  all  social  science  research,  this  natural  experiment  leaves  additional  room  for  explanation, 
exploration,  and  improvement.  Consequently,  we  encourage  future  research  on  the  effectiveness  of  blind 
pretesting.  As  this  study  is  perhaps  the  first  to  analyze  the  testing  effect  in  finance,  which  is  both  a 
quantitative  and  qualitative  subject,  we  posit  a  similar  blind  pretesting  approach  could  improve  student 
performance  in  the  hard  sciences.  Additionally,  a  more  longitudinal  or  broader  matched  sample  would  only 
provide  more  compelling  evidence— either  that  our  students  were  anomalous  or  that,  more  likely  in  our 
opinion,  there  is  merit  to  pretesting  in  a  corporate  finance  course.  Unfortunately,  with  our  instructor 
scenario,  which  includes  frequent  faculty  rotations  and  limited  terminal-degree  holding  instructors  with 
many  years  of  teaching  experience,  the  two  semesters  in  this  study  represents  our  best  opportunity  for  the 
foreseeable  future. 

Even  considering  these  potential  improvements  to  studying  the  efficacy  of  blind  pretesting,  as 
initial  skeptics  we  have  become  believers  through  this  analysis.  While  we  would  like  to  believe  the  results 
stemmed  from  a  significant  increase  in  teaching  performance  or  perhaps  an  effective  use  of  current  events, 
this  interpretation  is  only  self-serving  and  we  are  too  realistic  to  presume  this  is  the  case.  And  as  economists, 
we  weigh  the  certain  marginal  cost  of  time  spent  creating  the  pretest  and  the  15  minutes  spent  in  class 
administering  it  against  the  potential  marginal  benefit  of  improved  performance  suggested  in  this  study. 
Given  the  uncertain  but  apparently  positive  payoff,  particularly  for  the  lower  performing  students,  we 
intend  to  implement  this  tool  in  our  classes  going  forward. 


4  Carey,  Benedict,  “Wiry  Flunking  Exams  is  Actually  a  Good  Thing,”  New  York  Times  Magazine,  September  4,  2014, 
http://nvti.ms/lQzU2mk 
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